Intelligent sports video and data generation from ai recognition events

ABSTRACT

An intelligent sports video and data generating system using AI detection engines in Sports Detection Devices, broadcasting commands that incorporate global time stamp information to a plurality of the Sports Detection Devices, such that the recorded sports action data can be time-aligned with video data, wherein an automatically spliced together video or data set can be generated based on the parameters of an input query.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 63/030,688 filed on May 27, 2020; which is hereinincorporated by reference in entirety.

FIELD OF THE INVENTION

The present invention relates generally to aspects of detecting sportsactions using a Sports Detection Device having embedded therein anartificial intelligence sports recognition engine and methods to aligncaptured video data with detected sports actions or alternatively alignand compare tracked sports data received from multiple sources.

BACKGROUND OF THE INVENTION

Sports analytics is a space that continues to see a lot of growth. Inparticular, various artificial intelligence and machine learningprocesses are being utilized to ascertain a variety of new trackablestatistics including how far a player has traveled in a given game or ona given play. The amount of energy being exerted. Techniques regardingrunning, jumping, hitting, dribbling, shooting, skating and so forth.

Various wearable devices have been developed over the years to sense andgather data associated with various physical activities. For example, apedometer is one of the earliest forms of a wearable device that couldcalculate the number of steps an individual has taken. These deviceshave advanced with improved sensors, accelerometers, gyroscopes, heartrate monitors, and so forth.

Video is a powerful tool in sports. It provides a visual context foractions, motions and athlete/team performance. The medium of video isused at every level of enjoying sports and developing players and teams,from youth through college to the professional leagues. However therehas always existed a need to save time by reducing full-length videodown to just the key moments of interest. For example, this could be themoment a particular player is in action (e.g., skating on the ice) orthe moment a goal or basket is scored.

There is a desire to improve upon techniques of gathering, aligning andconveying data in multiple formats and from multiple sources in aplatform quickly while operating in a manner that is mindful of battery,memory, human and other resources.

The present application seeks to solve some of these identified problemsas well as other problems that will become apparent to those skilled inthe art.

SUMMARY OF THE INVENTION

The present application relates to an artificial intelligence (AI)sports recognition engine capable of identifying specific motions oractions of a sport or activity, coordinating the communication and timecoordination among the various sports detection devices used, as well astime-align sensed data indicative of sports action from the sportsdetection devices to be time-aligned with video associated with thesports session. Once aligned the intelligent system can thenautomatically generate a variety of edited videos based on an inputquery.

In one embodiment a method of coordinating time on a plurality of sportsdetection devices comprising the steps of broadcasting a global timestamp signal to a plurality sports detection devices; generating on eachsports detection device a session identification number based on thebroadcast global time stamp, wherein the session identification numberis associated with a sports session; generating a plurality of localtime stamps in response to the generated session identification number;receiving sensed data by the sports detection device and determiningwhether a sports action has occurred; and associating a local time stampwith each determined sports action.

The sports detection device used can be comprised of: a sensor arraysystem having a plurality of sensors and an AI recognition engine; atleast one CPU or MCU; memory, antenna; and a power source. The AIrecognition engine can be configured to receive sensed data from theplurality of sensors from an associated individual performing sportsactions and identify from the sensed data using an action detectionmodel a specific sports action.

Additional steps to the above embodiment can include receiving videodata associated with the sports session, wherein the video data includesa global time stamp. It can also include time aligning the video datawith each determined sports action. In some variations the method abovecan automatically edit the time aligned video to portions wherein adetermined sports action occurred. This can further includeautomatically editing the time aligned video further to portions whereina particular individual associated with the determined sports actionoccurred.

In some variations to the embodiment above a mobile computing device canbe used to send a connectionless signal indicating a timed-trial mode isinitiated to one or Sports Detection Devices.

Another set of steps that can be included in this method of coordinatingtime data on a plurality of sports detection devices comprises:monitoring by the sports detection device for the beginning of a sportsaction to be timed; identifying by the sports detection device theending of the sports action to be timed; and sending the timed sportsaction from the sports detection device to a computing device.

Another step can include generating one of the following commands:start, stop, update, timed-trial and annotation. In some variations thesports detection devices can receive from an annotation device (or amobile computing device) an annotation command.

The annotation command can further include inputting into a wirelessdevice one of an audio input, visual input, and pre-determined marker tobe associated with the local time stamp and configured to be retrievablelater with the associated sensed data at or around the moment of theannotation command.

The mobile computing device can continue to periodically broadcast asignal during the sports session that includes the original global timestamp sent and a local time offset. This allows any sports detectiondevice receiving the periodically broadcast signal after the originalbroadcasting of the global time stamp to then be associated with thesame sports session. The global time stamp can be in UTC form and thelocal time offset can be in 1 second increments from the original globaltime stamp.

In addition to coordinating time data on a plurality of sports detectiondevices, the system can receive multiple sets of video data each havinga global time stamp from a plurality of video sources associated withthe sports detection session. Then each set of video data can beautomatically time-aligned to the sensed data from the sports detectiondevice.

In another embodiment, once the sports sensed data is captured with aglobal time stamp and video data is also captured with a global timestamp, intelligent video editing methods and outputs can beaccomplished. For example, in one process of generating edited video thesteps can include: receiving an input query, identifying sensed databased on the query, automatically time-aligning the identified senseddata to the video data, and generating an output of the time-alignedidentified sensed data and video data in response to input query.

The input query can include any type of sports action, player, playsequence, sports session, and type of video data. The input query canalso include parameters about the type or style of output video, as wellas utilization of various video data. This input query can includeassociating a player with a player profile, wherein the player profileis associated with one or more sports detection devices.

In yet another embodiment a computer implemented method of generatingspliced sports video clips comprising the steps of: receiving an inputquery, wherein the input query includes at least one of: a request abouta player, a sports session, and type of sports action; identifyingsensed sports data based on the input query from a sensed sports datadatabase; identifying video clips based on the identified sensed sportsdata from a video data database, wherein each set of video dataassociated therewith has an associated global time stamp; automaticallysplicing the identified video clips into an edited video.

The computer implemented method of generating spliced sports video clipscan also utilize sensed sports data associated with a sensed sports datadatabase that is generated from a plurality of sports detection devices.Each sports detection device can be configured to determine when aparticular sports action has occurred.

Similar to the other embodiments, each sports device can be associatedwith a particular player profile during a sports session, and whereineach sports session is initiated by broadcasting a global time stamp.

This embodiment can further include the step of automatically annotatingthe edited video based on the input query. The annotating can include atleast one of type, length, and measured data associated with a givensports action. For example, the velocity of a slapshot or baseball hit,and the distance traveled.

In yet another variation to the embodiment the step of displaying thegenerated edited video is performed. This displayed video can then beused to analyze various sports actions to generate training and otherimprovement protocols and strategies.

In yet another embodiment an intelligent video and data generationsystem comprises: an interface configured to receive an input queryincluding parameters about a sports action; a sensed sports datadatabase, having sensed sports data generated from a plurality of sportsdetection devices, each sports detection device having an AI recognitionengine integrated therein and configured to receive a command includingglobal time stamp information; a video data database, wherein the videodata includes global time stamped information; and a processorconfigured to time align the sensed sports data and the video data basedon the global time stamp.

The processer can be used to identify relevant video data based on theinput query and the time-aligned video data and sensed sports data. Theprocesser can further generate a spliced video for viewing of therelevant video data. Similar to other embodiments the input query caninclude various parameters about video data and sensed sports detectiondata.

This intelligent video and data generation system can further includethe step of identifying and generating a video.

In yet another embodiment, a method of communicating with a plurality ofsports detection devices comprising the steps of: broadcasting from amobile computing device a command that includes a global time stamp aspart of the command to a plurality sports detection devices; generatingon each sports detection that received the command device a sessionidentification number based on the broadcast global time stamp, whereinthe session identification number is associated with a sports session;generating a plurality of local time stamps in response to the generatedsession identification number; sending an outbound broadcast from eachof the plurality of sports detection devices indicative that eachreceived the broadcast command from the mobile computing device;rebroadcasting the command periodically from the mobile computingdevice; and receiving by at least one additional Sports Detection Devicethe rebroadcast command or the outbound response, and generating theassociated session identification associated with the sports session onthe additional Sports Detection Device.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of theinvention will be apparent from the following description of particularembodiments of the invention, as illustrated in the accompanyingdrawings in which like reference characters refer to the same partsthroughout the different views. The drawings are not necessarily toscale, emphasis instead being placed upon illustrating the principles ofthe invention.

FIG. 1A illustrates a processing block diagram for an AI recognitionengine that uses sensor data input and produces a pre-definedrecognition result;

FIG. 1B illustrates a processing block diagram for the AI recognitionengine of FIG. 1A with further details on the inside of the recognitionengine;

FIG. 2 illustrates the process steps for generating a training model foruse with an AI recognition engine;

FIGS. 3A-B illustrate electronics block diagrams with an AI recognitionengine functional block for use with a Sports Detection Device;

FIG. 4A illustrates a smart wearable with an embedded AI sportsrecognition engine;

FIG. 4B illustrates an example placement and mounting location of thesmart wearable of FIG. 4A on protective shoulder pads;

FIGS. 5A-C illustrate various views of a smart hockey puck with anembedded AI sports recognition engine;

FIGS. 6A-B illustrate various individuals performing sports actionswhile using Sports Detection Devices having an AI recognition engineembedded therein;

FIG. 7 illustrates various components of a Sports Detection System;

FIGS. 8A-D illustrate various processing block diagrams for generatingan intelligent sports video from video source(s) and AI recognitionevents;

FIGS. 9A-E illustrate various commands broadcast to a plurality ofSports Detection Devices;

FIGS. 10A-E illustrate key elements of various broadcast commandsincluding Team ID information;

FIG. 11 illustrates a flowchart of an example workflow of broadcastingcommands to a plurality of Sports Detection Devices and receivingacknowledgement back;

FIG. 12 illustrates an interface showing where the video data and thesensed data are time-aligned, this interface can include the videoframe, data sample, recognition event and time stamp;

FIG. 13 illustrates a flowchart illustrating automatically generating aspliced video based on an input query; and

FIG. 14 illustrates a workflow where Sports Detection Devices canreceive commands directly or indirectly.

DETAILED DESCRIPTION OF THE INVENTION

Traditional methods of editing sports video for analysis and trainingpurposes can be cumbersome, take a lot of man resources, and time toproduce. One of the problems the inventors of this application areseeking to solve is a way to automate this traditional manual videoediting process through use of utilizing Sports Detection Devices thatinclude AI recognition engines to identify sports actions.

This sensed data once aligned with corresponding video data of the samesports session can allow for an intelligent input query system torapidly identify various paraments and generate a desired spliced oredited video based on the input query.

Another problem to be solved is how to communicate and ensure aplurality of Sports Detection Devices are configured in a manner suchthat they can be aligned quickly and consistently. Some of the solutionsinclude and more will become evident below is using a mobile computingdevice, such as a smartphone, to broadcast commands that include aglobal time stamp, which then can be used be each of the plurality ofSports Detection Devices to aligned and associated with a particularsports session. Other problems to be solved include ensuring that all ofthe desired Sports Detection Devices are receiving the various commands,and solutions for those are provide below. Again, these are a fewproblems and others are solved by the inventors of these embodimentsdescribed in the description below.

To provide clarity, the applicants would like to provide context aroundcertain terms used throughout this description that is in addition totheir ordinary meaning.

Artificial Intelligence (AI) recognition engine 100 is used to determinea sports action using a configurable processing node(s) that areconfigured to have machine learning or other AI methods or modelsencoded therein. In some variants, additional context-specific dataprocessing methods or models can also be encoded therein and be a partof the AI recognition engine. This AI recognition engine can be part ofa Sports Detection Device.

A Sports Detection Device can include a sensor array system, memory, aMCU or CPU, antenna, and power. The sensor array system can include oneor more sensors as well as the configurable processing node(s) that formpart of the AI recognition engine 100. It can be implemented intovarious wearable devices or other sports related equipment, such assmart hockey pucks. These devices can also include wirelesscommunication components for receiving and transferring data.

An Action Detection Model (ADM) can be a training model that can beencoded onto the AI recognition engine.

A secondary computing device can include a computing device with higherprocessing power and generally increased memory capacity over that of aSports Detection Device. Examples include tablets, laptops, desktopcomputers, cloud-computing and even smartphones.

Data Mining or Pattern Recognition methods can include variousalgorithms and techniques used to take tagged data, identify a patternassociated with the tagged data, so that additional data can be reviewedto identify other instances that are similar to the tagged data. Forexample, if the tagged data is indicative of a particular sports action,such as a skating stride or slapshot, the data mining and patternrecognition techniques can be used to identify other instances inrecorded data where another skating stride or slapshot has potentiallyoccurred.

A Supervised Learning Algorithm is configured to use to tagged data,other identified sports action data, parameterization inputs, falsesports action data, and profiler feedback to generate a training modelor Action Detection Model for use with the AI recognition engine. Thissupervised learning algorithm can consist of an outcome variable (ordependent variable) which is to be predicted from a given set ofpredictors (independent variables). Using these set of variables, it cangenerate a function that maps inputs to desired outputs. The trainingprocess continues until the model achieves a desired level of accuracyon the training data. Examples of Supervised Learning algorithms includeRegression, Decision Tree, Random Forest, KNN, Logistic Regression, etc.

Parameterization inputs can include various parameters includingminimums, maximums, statistical parameters, types of sensor data, foruse with creating the ADM.

Data tagging or tagged data includes identifying a specific sportsaction in the sensed data. This can be done by a profiler, who isreviewing time-aligned video and sensed data.

A profiler can be an individual who can identify a particular sportsaction, which can include a wide number of physical feats performed byan individual, such as an athlete. Sports action types include skating,shooting, hitting, throwing, jumping, running, blocking, dribbling, andso forth.

Sensed data can include data that is gathered by a Sports DetectionDevice and can include acceleration across multiple axes, rotationalmotion across multiple axes, magnetic field sensing across multipleaxes, temperature readings, pressure readings, impact readings, RFIDfeedback, signal feedback, and so forth.

Video data can include visually recorded data of an athlete performing aspecific sports action. Both sensed data and video data can includetimestamps for alignment.

A Sports Detection System can include one or more Sports DetectionDevices, one or more video recording devices, one or more secondarycomputing devices, and one or more profilers or any combination thereof.

A Sports Session can be a time period over which a set of sports actionsare performed, for example a practice, game, skills training, orcombination thereof.

Sports Session ID can be a unique identifier for a Sports Session thatdistinguishes it from other Sports Sessions.

Connectionless Broadcast can include the process of transmittinginformation wirelessly without forming a protocol connection. Forexample, in Bluetooth this is the process of wirelessly advertisinginformation without forming a direct wireless connection.

Broadcast Command can include a wirelessly transmitted action or statetransition to be performed by a Sports Detection Device.

Session Start Command can include a broadcast command to one or moreSports Detection Device(s) to start a Sports Session.

Session Update Command can include a periodic broadcast command to oneor more Sports Detection Device(s) to maintain the status and timing ofthe Sports Session.

Session Stop Command can include a broadcast command to one or moreSports Detection Device(s) to stop a Sports Session.

Session Annotation Command can include a broadcast command to one ormore Sports Detection Device(s) to associate a specific annotation to amoment in time.

Timed-Trial Command can include a broadcast command to one or moreSports Detection Device(s) to perform a time measurement for a sequenceof sports actions.

Time Coordination can include a process of establishing a global timingamongst one or more Sports Detection Device(s) that can be used foraligning events detected on Sports Detection Device(s) in the future.

Time Alignment can include a process of taking multiple data sources(e.g., recognition events for Sports Detection Device and video fromvideo source(s)) and synchronizing data based on one or more global timestamps.

Global Time Stamp can include a common point in time that isreferenceable based on a common time system (e.g., Coordinated UniversalTime or UTC).

Session Seconds Offset can include a time counter that is incrementingthroughout a Sports Session from the start of the Sports Session to theend of the Sports Session.

Sports Coordination Device can include a device used in the operation ofa Sports Session. For example, this could include a scorekeeping device,a timekeeping device, and a video capture device.

Various Sports Action Detection Methods will be further described belowand can implement many of the items noted above as well as varioussteps.

As semiconductor sensing technology matures there are increasingadvancements for integrating dedicated processing nodes close tosemiconductor sensing elements. These processing nodes are configurableto be encoded with machine learning methods and other artificialintelligence (AI) methods. Traditional smart or embedded products thatcan sense or measure motions of a sport or activity suffer from memorylimitations whereby an on-board application processor records data fromsensors and possibly implements some algorithm(s) (e.g., digitalfilter), but ultimately these products are limited by on-board memory orprocessing limitations. The memory limitations typically result inrequirements to maintain connectivity or proximity to a mobile device(e.g., smart phone, tablet) or data link (e.g., Bluetooth, Wi-Fi, LTE)in order to not exceed on-board memory. The processing limitationsresult in limited on-board algorithmic capabilities which in turn limitsoverall functional intelligence of such traditional smart or embeddedproducts.

However, some of the embodiments herein utilize the integrated,dedicated and configurable processing nodes close to semiconductorsensing elements to solve the limitations noted above. These processingcapabilities can be configured to implement training models foridentifying specific sports actions. By integrating this level offunctional intelligence into a smart product, a Sports Detection Deviceis realized, and large amounts of sensor data can be reduced to asubstantially smaller number of pre-defined recognition outputs, freeingup valuable resources of the on-board application processor. The smallernumber of pre-defined outputs is more suitably stored in on-boardmemory, and the dedicated processing node off-loads the primary on-boardapplication processor (e.g., CPU/MCU) which reduces the dependence ofthe product on outside devices or data links to circumvent on-boardmemory or processing limitations. This also can increase battery life ofthe Sports Detection Device. The Sports Detection Device implementsAction Detection Models that can determine multiple types of sportsactions.

An example of sensor or sensor array 110 configured to detect multipletypes of inputs is shown in FIG. 1A and FIG. 1B from sensors having 3axis of acceleration inputs and 3 axis of rotational velocity inputs. Ifthe main application processor were powerful enough it could do morecomplex analysis onboard, but then limitations in power from a batterysource become a limiting factor. Thus, as described in part above, anefficient and effective ADM is needed to compensate for the limitationsof onboard memory, required connectivity or proximity to a mobile deviceor required data link, processing and power for a sensing device.

For purposes of this application Sports Detection Devices or smartdevices can be integrated into sports equipment such as pucks, balls,and bats (some examples shown in FIGS. as well as into wearable devices(an example shown in FIG. 4A-B) that can be worn by a player orintegrated into gear worn by a player including jerseys, pads, helmets,gloves, belts, skates and so forth. The Sports Detection Devices areconfigured to capture data associated with a motion or action associatedwith a player, such as the data associated with a skating motion oraction of an ice hockey player.

The sensor array 110 can capture data such as acceleration, rotationalvelocity, radar signature, RFID reads, pressure and temperaturereadings. The data can be stored in the memory and later transferred toa secondary computing device (700, 710). The secondary computing devicemay be a laptop computer, a desktop computer, a local server, a smartphone, a tablet, or a cloud server, such as shown in FIG. 7 . The datacan also be pre-processed, analyzed or filtered utilizing the ADM priorto storing in memory to utilize the capabilities of the ADM to reducememory footprint.

In one embodiment, sensor data is captured by the sensor array and sentto the artificial intelligence (AI) recognition engine that includes anADM to determine a sports action performed by the player, such as askating action. FIG. 1A illustrates a processing block diagram for theAI recognition engine 120 that uses sensor data input and produces apre-defined recognition result 130. The pre-defined recognition results130 can be categorized into various specific sports actions, such asshown in FIG. 1A, but not limited to: skating detection, stridedetection, slapshot detection, wrist shot detection, snap shotdetection, backhand shot detection, stick handling, pass detection,board impact detection, goal impact detection, save detection, restdetection, being-checked detection, and so forth.

FIG. 1B illustrates the processing block diagram of FIG. 1A with furtherdetails on the inside of the AI recognition engine 120. The sensor datareceived from the sensor array 110 may include acceleration, rotationalvelocity, magnetic field strength, radar signature, RFID reads, pressureand temperature. The sensor data is then mapped as one or more signalsinto one or more processing blocks that produce one or more parameteroutputs in the AI recognition engine 120. For example, the accelerationsensor data could enter into processing blocks that include adifferentiator, an integrator, or a double integrator. Theses processingblocks would produce parameters such as jerk, velocity, and position ofthe sensor respectively. The rotational velocity sensor data could enterinto other processing blocks that include an integrator, adifferentiator, and a double differentiator. These processing blockswould produce parameters such as position, rotational acceleration, androtational jolt of the sensor respectively. The same or additional datacan be entered into additional processing blocks to determine additionalparameters. The parameters are then processed and compared to the ADM(training model) 122 by a configurable processing node 126 to determinea sports action associated with the determined parameters over the timeperiod of interest. The configurable processing node 126 is set to matchspecific parameters or data with specific sports actions in the ADM. TheAI recognition engine results are improved by a context-specific dataprocessing model 124. The context-specific data processing model 124 canfunction as an additional layer to provide better accuracy to the ADM.For example, the context-specific data processing model 124 can providefixed boundaries or limitations for certain sports actions, whereas theADM might still consider those or not appreciate the order ofoperations. One specific example includes detecting skating strides. TheADM might detect sequential skating strides, and output right stride,left stride, left stride, left stride, right stride. Thecontext-specific data processing model 124 would recognize that there isa sequential order to the strides and override what the ADM perceived as3 left strides in a row to modify the middle left stride to a rightstride. Thus, in combination the ADM 122 and context-specific dataprocessing model 124 can more accurately output identified sports actionresults 130.

FIG. 2 illustrates an embodiment for a process 200 of generating orupdating an ADM (training model) 228 that is used by the AI recognitionengine 212. A Sports Detection Device 210 that is associated with anindividual is placed on or in sports equipment or gear and collects datausing the embedded electronics 214, which includes power, memory andsensor array, as well as the AI recognition engine 212. This collecteddata that can be raw sensor data or pre-filtered by the AI recognitionengine is sent to a secondary computing system 220 that can includeother processing devices, such as computers or cloud-based computingdevices. The collected data can then be tagged 222 for instances of aspecific sports action identified and data-mined 224 using the taggingto identify additional instances in the collected data of the sportsaction. This data tagging 222 and data-mining 224 output can then besent to a supervised learning algorithm 226 or machine learning or otherAI methods that generates or updates an ADM (training model) 228. TheADM (training model) 228 is then deployed and utilized to update the AIrecognition engine 212 onboard the Sports Detection Device 210 todistill the sensor data received to a specific sports action that isagain stored in memory and can then be sent again to secondary computingfor further refinement as noted. It should be noted that the datatagging can be performed by a profiler. The parameterization input canalso be performed by a profiler, user, or data-scientist. The datatagging can be aided by data mining and pattern recognition techniques,which help expedite the data-tagging process.

FIGS. 3A-B illustrates electronics block diagrams with an AI recognitionengine functional block 350, which can be integrated into a SportsDetection Device. As shown, in one configuration a sports detectiondevice electronic block 300A includes a power supply 310, microprocessor(MCU) or CPU 320, one or more sensors that can be part of a sensor arraysystem 340, memory 330, AI recognition engine 350 that is comprised ofprocessing nodes configured to run an ADM 122 and/or Context-SpecificData Processing Model 124, such as shown in FIGS. 1A-B, and antenna 360.As shown in 300A, 350 is integrated directly into the sensor arraysystem 340. Memory 330 can be optionally integrated with the CPU/MCU 320or configured separately. Alternatively, as shown in Sports DetectionDevice electronic block 300B, the AI recognition engine 350 can beintegrated with the CPU/MCU 320. However, integrating the AI recognitionengine directly into the sensor array system is preferable if itoffloads processing load, power consumption and demand from the CPU/MCU.The antenna 360 can be utilized to receive connectionless commands, forma connection to a secondary computing device and also transmitinformation, such as sensed data associated with a sports action. Theantenna 360 can be comprised of one or more types of antennae and beable to wirelessly communication across several types of wirelesssignals including BLUETOOTH, Wi-Fi, NFC, cellular and other types ofradio signals known in the art.

FIG. 4A illustrates a smart wearable or Sports Detection Device 400 withan embedded AI sports recognition engine. This device 400 can be placedor mounted in various locations including on protective shoulder pads410 worn by a hockey player.

FIGS. 5A-C illustrate various views of a smart hockey puck 500, which isanother form of a Sports Detection Device that can include an AIrecognition engine with an ADM embedded therein that is configured to begenerated and updated using the methods described herein.

FIGS. 6A-B illustrate various individuals/athletes using SportsDetection Devices 400 and 500 having an AI recognition engine embeddedtherein. In FIG. 6A the individual 600A can use the device 400 todetermine when a stride using skate 610A or 610B occurs. The skatingdata can be aligned with video and used later for analysis in coachingand training sessions, which is another purpose of acquiring accuratesports action data through the Sports Detection System and methodsdescribed herein.

FIG. 6B illustrates a hockey player 600B wearing a device 400 and alsousing a device 500 with hockey stick 610. When the ADM is appropriatelyembedded in the AI recognition of device 400 or 500 it will be able todetermine when a slapshot occurred as well as all the data associatedwith the given slapshot. Once aligned with video data, the system canproduce each slapshot for visual inspection as well as the correspondingdata associated therewith. For example, rotation, speed, contact timewith the blade of the hockey stick and so forth.

FIG. 7 illustrates various components of a Sports Detection Systemincluding, in this particular case, a smart hockey stick 610, smart puck500, which transmits information to a secondary computing device 700(here shown as a smartphone), which can further process and communicatewith another secondary computing device 710, such as cloud-computingresources.

FIG. 8A illustrates a processing block diagram 800A for generating anintelligent sports video from video source(s) 810 and AI recognitionevents 820. In this case the video source(s) 810 can include raw videoor sets of video segments that do not include additional intelligencederived from the video frames. This approach requires the least amountof additional processing resources on the video source(s) and leveragesthe intelligence from the time-stamped AI recognition events. Followingthe alignment step 830 the intelligent video generation step 840 is usedto output one or more particular video segments of interest. Forexample, the intelligent video generation step 840 can be used to splicea full-length sport video into just the segments in time when a playeris on the ice or playing field actively playing and remove the segmentsin time when the player is resting or on the bench. In this example, thetime-stamped AI recognition events from the smart wearable or SportsDetection Device 400 (e.g., stride detected, shot detected, etc.)provide the necessary information to splice or cut the full-length videointo the sub-segment of only playing time. These sub-segments can betagged with an AI recognition tag that is derived from the type of AIrecognition event detected, logged and communicated in the smart device.These sub-segments can be broken down in several ways including limitinginstances where a specific player takes a slap shot, or all theinstances players on a particular team took slap shots, instances ofsaves for an individual or team, and various other filtered requeststhat pertain to isolating and compiling video surrounding specificsports action that have been detected using a Sports Detection Device.If multiple video sources, each using varying angles or zoom levels,then those could also be spliced together to show the various views of aparticular sports action back-to-back or alternatively spliced to beviewed in the form of grid or other arrangement showing the multipleviews simultaneously. Certain sports actions can even have the actualtime of the sports action shown in video automatically slowed down toillustrate and analyze the mechanics of the given sports action. Variousother editing techniques can be implemented for the output, many ofwhich are enabled once the sensed sports action data becomes alignedwith video data of the particular event.

One of the processing shown in FIG. 8A is that, particularly in the caseof team sports, the video sub-segments can be known to accuratelyinclude the player of interest, but there will likely be other playersor actions in the video frames that the observer/viewer must filter outto correctly identify the player or individual of interest.

FIG. 8B expands on FIG. 8A and illustrates a processing block diagram800B for generating an intelligent sports video from video source(s) andAI recognition events with a pre-alignment video algorithm 812 appliedto the video source(s) 810 before the alignment step 830 with AIrecognition events 820. In this embodiment 800B the post-processed videosource, rather than simply the raw video source, is merged with thetime-stamped AI recognition events 820 from the smart device to createadditional capabilities to add intelligence to the video. One advantageof adding this additional video processing step before the alignmentstep is that additional intelligence can be derived from the sourcevideo. For example, the pre-alignment video processing algorithm canapply a deep learning approach like human pose estimation, which is usedto determine, identify and track the human joints (key points) of allthe players or individuals in a video frame. In a single video frame,where multiple players are shown and/or moving, this information can beused to track the location of the players or individuals on the ice orplaying field.

There are also varying levels of tracking that can occur fromframe-to-frame. For example, by locating the key joints (i.e., hip,knee, ankle) of a set of hockey players on the ice during a practice orgame and tracking the relative movement of those joints fromframe-to-frame, a video algorithm can generate its own AI recognitionevents (e.g., player taking a stride, player taking a shot, goaliemaking a save, etc.). This process generally is time-consuming and canuse a lot of processing resources. By identifying joints, for example,and coupling that with known players in a given video frame, theassociation of the player to the identified joints can more readily beassociated, thus reducing processing time. In particular, if a givenplayer in a video frame is the player taking the identified slapshotthen overlaying that AI sensed information and associated player profilewith that player's identified human joint profile can be used to isolatethe player in that video frame as well as other analyzed video frameswhere multiple players or persons exist. Thus, reducing some of theprocessing time needed by the video processing algorithm.

As noted, the additional video processing step in FIG. 8B allows forincreased options and capabilities after the alignment step during theintelligent video generation step. For example, when aligning the outputof the video pre-alignment algorithm, which includes video-derivedtimestamped AI recognition events, with the AI recognition events from asmart device, in addition to splicing or cutting the full-length videointo sub-segment containing a given player or individual of interest,those sub-segments can be automatically labeled with a visual markerindicating one or more players or individuals of interest. The visualmarker could be a pointer, highlighted portion, outline, or other typeof visual identifier. In this example, the person viewing theintelligently derived video is given a clear indication of where theyshould focus their eyes and attention.

The relative tradeoffs of the processing approaches shown in FIG. 8A andFIG. 8B relate to the amount of processing time and computing resourcesavailable for an application. For example, the processing shown in FIG.8A does not require a high-performance camera as the video source(s) anddoes not require the video to be processed by a computer vision(CV-)-based deep learning (DL) algorithm before the alignment step. Thisapproach is computationally inexpensive, can leverage existing camerasthat are most commonly available and depends on the intelligence of theAI recognition events generate inside one or more Sports DetectionDevice(s) to generate a more intelligent sports video. Comparatively,the processing shown in FIG. 8B does require higher performance cameraand computing resources to create one or more video source(s) that aresuitable to run a pre-alignment algorithm. The processing in FIG. 8Atherefore lends itself to more real-time or during practice/gameplayapplications, while the processing in FIG. 8B lends itself topost-practice/game applications. There are some cases where the videopre-alignment algorithm shown in FIG. 8B is performed after thealignment step, such as shown in FIG. 8C. 800C is an example of havingthis type of algorithm being applied post alignment 832. FIG. 8D showsthat there is the possibility of a pre-alignment algorithm of the event812, and then a post-alignment algorithm 832, this in turn leads to thegeneration of a video segment. 800D is the ability to have pre and postalignment algorithms produce an intelligent video generation.

FIGS. 9A-E illustrate various commands made via a connectionlessbroadcast to a plurality of Sports Detection Devices. In FIG. 9A a startsession command and timestamp are wirelessly transmitted from a mobilecomputing device (i.e. smartphone, tablet, laptop) to one or more SportsDetection Devices (400, 500) that are capable of generating AIrecognition events. As noted above, these can include smart wearable(400) or interactive Sports Detection Device (500), such as a smarthockey puck, Sports Detection Devices. The function of the start commandis to indicate to the Sports Detection Devices that a new sports session(e.g. practice, game, training session) has begun. This allows theSports Detection Device to achieve lower power levels and save memorystorage by not actively recording or analyzing until a sports sessionhas commenced indicated by the received Session Start Command. EachSports Session has an associated Sports Session ID that is a uniqueidentifier of the given sports session. This unique identifier can bebased on a global time stamp. The function of the global time stamp isto give the Sports Detection Device an updated time reference at thestart of the session. In many cases, the Sports Detection Device may nothave a global time reference, and thus can only track local time sinceits last power-up. In order to provide a global time reference to theSports Detection Device which can be used for time-stamping the AIrecognition events, the mobile computing device plays a key role ofsharing a global time reference. With this global time reference, theSports Detection Device can then assign timestamps to its generated AIrecognition events that will later align with the timestamps of one ormore video source(s), as described above.

There are many instances where not all Sports Detection Devices receivethe session start command. This could be the result of not being turnedon, delayed entering of the facility, an issue with receiving thesession start command broadcast due to interference or signal strengthand so forth. Thus, to ameliorate this concern, and as shown in FIG. 9Ba session update command can be broadcast periodically. This sessionupdate command can include the original global time stamp from thesession start command as well as a time offset. The offset can be in theform of second increments or other increments. For example, if a playerassociated with a wearable Sports Detection Device joins a practice 5minutes late, then when the associated Sports Detection Device receivesthe session update command it could include the original global timestamp, plus a 300 second local time offset. The original global timestamp will help the Sports Detection Device to generate the appropriateSports Session ID, while the offset will help the sensed data retrievedto be aligned according to the global time stamp. This then ensures thateach of the Sports Detection Devices throughout the sports sessionmaintain appropriate timing. In some cases, the session update commandcomes from Sports Detection Devices that were already included in thesession and they serve to assist a late-arriving device to also join thesession with appropriate time reference.

FIG. 9C illustrates a session stop command being broadcast to each ofthe Sports Detection Devices. The session stop command indicates to eachof the Sports Detection Devices that the given sports session has ended.The Sports Detection Devices can then go into a lower power mode ormemory conservation mode. The stop session command can also include theoriginal global time stamp and a local time offset information, but thatis optional. In some first cases Sports Detections Devices automaticallystop sessions after a timeout period when the device no longer sees AIrecognition events from sports actions. In some second cases, SportsDetection Devices automatically start sessions when AI recognitionevents from sports actions are observed and only save those events tomemory upon receipt of a stop session command. Lastly in some thirdcases, Sports Detection Devices automatically start and stop sessionsbased on AI recognition events from sports actions. All three casesprovide different levels of simplifying the number of steps that arerequired of a user of the computing device.

FIG. 9D illustrates a session annotation command being broadcast to eachof the Sports Detection Devices either from a mobile computing devicelike the other commands or through a sports coordination device. Asnoted above, the sports coordination device can include any number ofdevices used in the operation of sports session, which includescorekeeping devices, timekeeping devices, other statistical capturingdevices, and video capture devices. One of the primary purposes of theannotation command is to associate a specific annotation with a momentin time during the sports session. For example, if a player scores, theannotation can include that a score was made at a particular time. Thiscan then be used as another filtering mechanism when generatingintelligent video segments, such as splicing together all video sourcesfrom 3 seconds before until 2 seconds after the score. These annotationscould include coaching annotations, which could be any number ofcomments that the coach wants to readily ascertain or later review, forexample a positive pass or a poor pass. An example of video capturedevices could include a particular video capture device islive-streaming versus simply recording. Another example includes anannotation of the video capture device location. Again, each of thesetypes of annotations can be used as part of an input query to helprapidly sort and intelligently compile a video based on the input queryparameters.

FIG. 9E illustrates yet another type of command being broadcast, whichis a time trial command. This time-trial command can put the SportsDetection Device on notice to capture the beginning and ending of aseries of sports actions and associate the time therewith. For example,partway through a hockey practice, a coach wants to have each of theplayers engage in a time-trial of seeing how fast they can skate to oneend of the rink and back. With the time-trial command, all players canbe timed parallel, but the players do not have to all simultaneouslystart at the same time. They can start when they are ready, because theAI recognition on each of the Sports Detection Devices will identifywhen the given player starts their sprint and when they ended it basedon the sensed data received into the Sports Detection Device. Thoseresults can then be immediately sent to a device such as the same mobilecomputing device the coach used to initiate the time trial command fordisplay or retrieved at a later point. Time trial commands can includethe option of immediately broadcasting the results from the SportsDetection Device without having to send the additional sports actiondata retrieved during the sports session.

FIGS. 10A-E illustrate broadcast commands similar to those of FIGS. 9A-Eincluding additional Team ID information. In some embodiments, theSports Detection Devices are configured to broadcast a return signalindicating that they received the command broadcast. In the instancewhere the broadcast command includes a command code, sports session ID,time offset and team ID, the Sports Detection Device can determinewhether the broadcast command applies to the given Sports DetectionDevice and if so, respond accordingly.

FIG. 11 illustrates a flowchart of an example workflow of broadcastingcommands to a plurality of Sports Detection Devices. A user, such as acoach, can generate a command code to be broadcast to a plurality ofdevices during step 1110. This can be generated on a user interfacerunning on an application on a mobile computing device such as asmartphone. Once the broadcast command is generated then it can bebroadcast out to all the Sports Detection Devices in the area. In theinstance, multiple teams or groups are practicing or playing in the samevicinity, the broadcast command can include a Team ID or identificationnumber associated with it. Once a given Sports Detection Device receivesthe inbound command during step 1120 it can then determine what to dowith it. Initially it can determine if the Sports Detection Device isassociated with the broadcast Team ID 1122. If so, then it can proceedto process the additional information including the received commandcode 1124, store or reconfirm the session ID 1126, and maintain orconfirm the appropriate time offset 1128, which as noted above can be anoffset from a global time stamp that is associated with the session ID.After that is the processed by the Sports Detection Device, an outboundbroadcast 1130 from each Sports Detection Device can be sent withinformation indicating it received the given command or in someinstances ignored it. Similarly, the mobile computing device receivingthe outbound signal 1130 can confirm what it does with the information.Namely, does it have the same Team ID that was broadcast and if notchoose to ignore the broadcast. The team ID can be integral tocommunicating via an open broadcasting manner with regards to when theassociated devices should take action or not in an area with multipleteams so as to not interfere with another sports session.

As noted, an important aspect of the approaches described above is thatthe control and communication of the Sports Detection Devices by themobile computing devices can be accomplished without forming aconnection to any one or more devices. This connectionless approach forcommand and control is important because often mobile computing deviceshave limitations on the maximum number of connections that can be formed(e.g., 15 devices maximum). When managing a team sport like hockey therecould be 20 players using wearables and another 10-20 smart hockeypucks. By using a connectionless approach for command and control thesystem can support a large number of devices that is much greater thanany limitation imposed by a specific mobile device.

As noted above, video data or other source data can be time-aligned withthe sensed sports action data received from the Sports Detection Devicesnoted. FIG. 12 illustrates an interface showing where the video data andthe sensed data are time-aligned. The sensed data can include abeginning and ending marker of a given sports action. These markers canbe expanded in a manual mode if the user prefers an expanded clip,shortened clip or offset clip of the action. Additional indicators canbe present such as the type of sports action performed, the playerprofile associated with the sports action, speed, direction, rotationand other measured statistics can also be shown. For example, thevelocity of a given slapshot and the time of the hockey puck spent onthe hockey stick during the slapshot, the given player's averagevelocity and other advanced statistics and measurements.

It should be well understood that in addition to video sources, audiosources could likewise be substituted in place, as well any other typeof globally time stamped captured information. Once time-aligned amyriad of intelligent splicing and outputting of information can begenerated as noted by several examples above. One interesting caseincludes time-aligning sensed data from two different player profiles.For example, if one hockey player checks another hockey player, what isthe impact force on each. This information can be time-aligned withoutvideo or audio data, but other sensed data.

FIG. 13 illustrates a method of automatically generating intelligentvideos of sports actions. In step 1310 a user enters an input query.This query can include various sorting or analysis parameters such as:type of sports action, player profile, length of sports action, numberof sports action performed in a session, footage before or after aspecific sports action, comparison of sports actions between two or moreplayers during one or more sports sessions, sports actions with aparticular attribute (such as 10 fastest slapshots or 10 slowestslapshots), sequential sports actions, specific or group of sportssession, location of sports session, sports sessions involvingparticular players, annotation information, timed-trial information andso forth. Once the query is input the computer can process the requestto search and identify 1320 in a database containing sensed sportsaction data provided by a plurality of Sports Detection Devices over aplurality of sports sessions. Once the identified results from thesensed sports database are obtained, the next step 1330 can includesearching the video database for video data that has the same globaltime stamped information as the identified sensed sports data. Thesesensed data and video data can then be intelligently time-aligned andspliced together in step 1340. It should be noted that the input querycan also include parameters about how and what type of video data isused. For example, if multiple video sources are identified for a givensensed sports action, then input query can guide which set is used, ifboth or used, if they used sequentially or combined simultaneously in aside-by-side or grid view. Other editing outputs should be readilyunderstood by those in the art, once the sensed data and video havebecome aligned.

FIG. 14 illustrates a workflow where Sports Detection Devices canreceive commands directly or indirectly. In this workflow, a mobilecomputing device broadcasts an original command in step 1410. This hasbeen described in various ways above. That broadcast may only bereceived by a subset of the intended plurality of Sports DetectionDevices in step 1420. Once the command is received, that subset ofSports Detection Devices can then broadcast in response to the originalcommand the appropriate outbound response in step 1430. In the interimthe computing mobile device can rebroadcast the original command in anupdated command that is broadcast in step 1440. Now the remaining SportsDetection Devices that haven't already received the original command caneither be notified from the updated broadcast command or alternatively,receive the outbound response, which is indicative of what the originalcommand requested as well as the latest time offset and thus begin torespond appropriately. This multi-approach communication method can helpwhen the mobile computing device goes into a low power mode and nolonger rebroadcasts every second, or when certain Sports DetectionDevices are in areas that are out of reach due to distance, interferenceor simply arriving late or being powered on late. Thus, ensuring thateach of the Sports Detection Devices is running as expected during asports session, because of the direct or indirect method of broadcastsfrom the mobile computing device or other Sports Detection Devices.

While the principles of the invention have been described herein, it isto be understood by those skilled in the art that this description ismade only by way of example and not as a limitation as to the scope ofthe invention. Other embodiments are contemplated within the scope ofthe present invention in addition to the exemplary embodiments shown anddescribed herein. Modifications and substitutions by one of ordinaryskill in the art are considered to be within the scope of the presentinvention.

What is claimed:
 1. A computer implemented method of generating splicedsports video clips comprising the steps of: receiving an input query,wherein the input query includes at least one of: a request about aplayer, a sports session, and type of sports action; identifying sensedsports data based on the input query from a sensed sports data database;identifying video clips based on the identified sensed sports data froma video data database, wherein each set of video data associatedtherewith has an associated global time stamp; automatically splicingthe identified video clips into an edited video.
 2. The computerimplemented method of generating spliced sports video clips of claim 1,wherein the sensed sports data associated with sensed sports datadatabase is generating from a plurality of sports detection devices,each sports detection device configured to determine when a particularsports action has occurred.
 3. The computer implemented method ofgenerating spliced sports video clips of claim 1, wherein each sportsdevice is associated with a particular player profile during a sportssession, and wherein each sports session is initiated by broadcasting aglobal time stamp.
 4. The computer implemented method of generatingspliced sports video clips of claim 1, further comprising the step ofautomatically annotating the edited video based on the input query. 5.The computer implemented method of generating spliced sports video clipsof claim 4, wherein the annotating can include at least one of type,length, and measured data associated with a given sports action.
 6. Thecomputer implemented method of generating spliced sports video clips ofclaim 1, further including the step of displaying the generated editedvideo.
 7. An intelligent video and data generation system comprising: aninterface configured to receive an input query including parametersabout a sports action; a sensed sports data database, having sensedsports data generated from a plurality of sports detection devices, eachsports detection device having an AI recognition engine integratedtherein and configured to receive a command including global time stampinformation; a video data database, wherein the video data includesglobal time stamped information; and a processor configured to timealign the sensed sports data and the video data based on the global timestamp.
 8. The intelligent video and data generation system of claim 7,wherein the processer can identify relevant video data based on theinput query and the time-aligned video data and sensed sports data. 9.The intelligent video and data generation system of claim 8, wherein theprocesser can further generate a spliced video for viewing of therelevant video data.
 10. The intelligent video and data generationsystem of claim 7, wherein the input query further includes parametersabout video data.
 11. The intelligent video and data generation systemof claim 7, further including the step of identifying and generating avideo.
 12. A method of communicating with a plurality of sportsdetection devices comprising the steps of: broadcasting from a mobilecomputing device a command that includes a global time stamp as part ofthe command to a plurality sports detection devices; generating on eachsports detection that received the command device a sessionidentification number based on the broadcast global time stamp, whereinthe session identification number is associated with a sports session;generating a plurality of local time stamps in response to the generatedsession identification number; sending an outbound broadcast from eachof the plurality of sports detection devices indicative that eachreceived the broadcast command from the mobile computing device;rebroadcasting the command periodically from the mobile computingdevice; and receiving by at least one additional Sports Detection Devicethe rebroadcast command or the outbound response, and generating theassociated session identification associated with the sports session onthe additional Sports Detection Device.